Regularization, Boosting and Mirror Averaging

In their paper, Peter Bickel and Bo Li give an interesting unified view of regularization methods in statistics. The literature on this subject is immense, so they outline a general conceptual approach, and then focus on some selected problems where regularization is used, such as regression and classification, or more generally, prediction. In this context, they discuss in detail a number of recently emerging techniques, in particular, boosting, estimation of large covariance matrices, estimation in the models where the dimension is larger than the sample size. It is difficult to overestimate the importance of regularization in statistics, especially in nonparametrics. Most of nonparametric estimation problems are ill-posed, and common estimators (kernel, histogram, spline, orthogonal series etc.) are nothing but regularized methods of solving them. The corresponding regularization parameters are just smoothing parameters of the estimators. The main ideas of statistical regularization can be very transparently explained for prediction problems. Assume that X1, . . . , Xn are i.i.d. observations taking values in a space X , and assume that the unknown underlying function f ? that we want to estimate belongs to a space F . Consider a loss function Q : X × F → IR and the associated prediction risk R(f) = EQ(X, f) where X has the same distribution as Xi. Assume that f ? is a minimizer of the risk R(f) over F . Then a classical, but not always reasonable, estimator of f ? is a minimizer over f ∈ F of the corresponding empirical risk